Communication support apparatus and method

ABSTRACT

An input unit inputs a drawing information based on a first language on a display by at least one of a plurality of users each locating at predetermined position relative to the display. A conversion unit converts the drawing information based on the first language to an information based on a second language. A control unit presents the information based on the second language on the display according to a location of another user of the plurality of users as a person to present the information based on the second language.

This is a division of application Ser. No. 10/294,819, filed Nov. 15,2002 now U.S. Pat. No. 7,277,845, which is incorporated herein byreference.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority fromthe prior Japanese Patent Application PH2001-357949, filed on Nov. 22,2001; the entire contents of which are incorporated herein by reference.

FIELD OF THE INVENTION

The present invention relates to a communication support apparatus and amethod for translating language information used among a plurality ofusers and for effectively presenting a translation result to a partnerin the plurality of users.

BACKGROUND OF THE INVENTION

In recent years, the opportunity for ordinary persons who go abroad isincreasing, and the number of people who live in a foreign country isalso increasing. Furthermore, a progress of communication technology orcomputer networking technology such as an Internet is also remarkable.Therefore, since an opportunity to communicate in a foreign language andan opportunity to interact with a foreigner has increased, a need forexchange between different cultures or between different languages hasbeen increasing. This is an inevitable flow because the world isbecoming borderless, and it is thought that this tendency continues tobe accelerated.

For the purpose of exchange messages cut across different languages orbetween different cultures, the necessity of communication in differentlanguages for persons whose native languages are different, or thenecessity of communication between different cultures for persons whosecultural background are different, are increasing.

As a method for communicating with one person who speaks a differentlanguage from another person's native language, one person can learnanother person's native language, or use a translator who can mutuallytranslates between a plurality of languages, can be considered.

However, learning a foreign language is not easy for anyone, andrequires great time and costs. Furthermore, even if a particular personcan master one foreign language, his partner may not be able to use thelanguage which the particular person has mastered. In this case, it isnecessary for the particular person to master a second or third foreignlanguage, and the resulting difficulty of language learning increases.

Moreover, since the translator is a professional with special skill, thenumber of the translators is also limited, and the employment costs arealso high. As a result, generally it is seldom used.

When ordinary people travel abroad, they often use a book withconversation phrases normally encountered with its translation. In thisbook, various kinds of expressions of fixed form phrase forcommunication are recorded.

However, since the number of conversation phrases in the book isrestricted, all expressions needed in an actual conversation cannot becovered, and the collection of conversation phrases is inadequate.Moreover, it is very difficult for a user to memorize the fixed formphrase recorded in the collection of conversation phrases in the sameway of learning of foreign language. Since the collection ofconversation phrases is a form of the book, it is difficult for a userto quickly discover the page in which a necessary expression isdescribed in the scene of an actual conversation. Therefore, in theactual communication, it is not necessarily effective.

Alternatively, an electronic translator as a portable size in which thecollection of conversation phrases is electronically maintained may beused. A user may use the electronic translator by hand and specifies asentence to be translated by a keyboard or a menu selection operation.The electronic translator changes an input sentence into a foreigncountry language, and outputs the translated sentence through a displayor a loudspeaker. In this way, communication with a partner who speaks adifferent language is performed.

In the electronic translator, the time and effort to retrieveconversation phrases necessary for a user is minimally reduced whencompared with the book (collection of conversation phrases). However, inthe same way of the book, the electronic translator stores the limitedfixed form phrases and some extended expressions which are partiallytransformed only from the fixed form phrases. In short, sufficientcommunication between persons each of which native foreign language isdifferent cannot be enabled.

Moreover, if the number of conversation phrases of the electronictranslator increases, the user's operation of the electronic translatoris executed by a keyboard and a menu selection. As a result, it isdifficult for the user to select a sentence to be translated. In short,validity in actual communication falls.

Thus, in the prior art, it is a problem that there are no tools tosufficiently support communication among a plurality of persons whoseuse languages are different.

SUMMARY OF THE INVENTION

It is an object of the present invention to provide a communicationsupport apparatus and a method of enabling smooth communication among aplurality of persons whose use languages are different.

According to the present invention, there is provided a communicationsupport apparatus, comprising: an input unit configured to input adrawing information based on a first language for presentation on adisplay by at least one of a plurality of users each locating atpredetermined position for the display; a conversion unit configured toconvert the drawing information based on the first language to aninformation based on a second language; and a control unit configured topresent the information based on the second language on the displayaccording to a location of another user of the plurality of users as aperson to present the information based on the second language.

Further in accordance with the present invention, there is also provideda communication support apparatus, comprising: an input unit configuredto receive an input message based on a first language from a user; atranslation unit configured to translate the input message into anoutput message based on a second language for another user; and apresentation unit configured to present the output message allocatedtowards a location of another user.

Further in accordance with the present invention, there is also provideda method for supporting communication, comprising: inputting a drawinginformation based on a first language for presentation on a display byat least one of a plurality of users each locating at predeterminedposition for the display; converting the drawing information based onthe first language to an information based on a second language; andpresenting the drawing information based on the second language on thedisplay according to a location of another user of the plurality ofusers as a person to present the information based on the secondlanguage.

Further in accordance with the present invention, there is also provideda method for supporting communication, comprising: receiving an inputmessage based on a first language from a user; translating the inputmessage into an output message based on a second language for anotheruser; and presenting the output message allocated towards a location ofanother user.

Further in accordance with the present invention, there is also provideda computer program product, comprising: a computer readable program codeembodied in said product for causing a computer to supportcommunication, said computer readable program code having: a firstprogram code to input a drawing information based on a first languagefor presentation on a display by at least one of a plurality of userseach locating at predetermined position for the display;

a second program code to convert the drawing information based on thefirst language to an information based on a second language; and a thirdprogram code to present the drawing information based on the secondlanguage on the display according to a location of another user of theplurality of users as a person to present the information based on thesecond language.

Further in accordance with the present invention, there is also provideda computer program product, comprising: a computer readable program codeembodied in said product for causing a computer to supportcommunication, said computer readable program code having: a firstprogram code to receive an input message based on a first language froma user; a second program code to translate the input message into anoutput message based on a second language for another user; and a thirdprogram code to present the output message allocated towards a locationof another user.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing the communication support apparatusaccording to the first embodiment of the present invention.

FIG. 2 is a schematic diagram showing an example of the contents of adisplay information memory 16 in FIG. 1 in an embodiment of the presentinvention.

FIG. 3 is a schematic diagram showing a position and an uprightorientation of a character drawn on a display screen in an embodiment ofthe present invention.

FIG. 4 is a schematic diagram showing an example of a Japanese characterdrawn on the display screen in an embodiment of the present invention.

FIG. 5 is a schematic diagram showing an example of the contents of alocation information memory 17 in FIG. 1 in an embodiment of the presentinvention.

FIG. 6 is a schematic diagram showing a location and an uprightorientation of a user for the display screen in an embodiment of thepresent invention.

FIG. 7 is a flow chart for explaining operation of the first embodimentof the present invention.

FIG. 8 is a schematic diagram for explaining a concrete example in thefirst embodiment of the present invention.

FIGS. 9A-9F is schematic diagrams showing transition of the displayscreen in the first embodiment of the present invention.

FIGS. 10A and 10B are schematic diagrams showing one example of theJapanese character and a symbol drawn on the display screen in anembodiment of the present invention.

FIG. 11 is a block diagram showing the communication support apparatusaccording to the second embodiment of the present invention.

FIG. 12 is a block diagram for explaining actual arrangement ofmicrophones 21 a and 21 b, and loud speakers 22 a and 22 b in FIG. 11 inan embodiment of the present invention.

DETAILED DESCRIPTION OF THE EMBODIMENT

Hereafter, various embodiments of the present invention will beexplained by referring to the drawings.

FIG. 1 is a block diagram showing the communication support apparatusaccording to the first embodiment of the present invention.

In embodiments of the present invention, in view of the development ofspeech recognition techniques and automatic recognition techniques ofhandwritten characters, and a background of improvement of speed andprecision of natural language processing techniques, such as a machinetranslation for a text, communication between different languages can besmoothly executed using above-mentioned technique.

In short, in embodiments of the present invention, after receiving aninput message from a user, the input message is translated so thatanother user to whom the input message is provided can understand theinput message. A presentation orientation of a translation message isdetermined according to the user's location. As a result, communicationamong a plurality of users whose use languages are different can beeffectively supported.

When communicating with a foreigner who speaks a foreign countrylanguage, a conversation of writing, which is conversation with apartner by writing a figure and a character using a pen, a pencil, or amemo pad, is often used. In this method, by displaying a picture or animage in handwriting, an effective communication with a partner can beperformed. However, it may be difficult to communicate with the partneronly by the picture or the image to tell things except for expressiveconcrete things. Additionally, since the cultural backgrounds of thepeople on both sides (a particular person and a partner) to communicateare different, a symbol or a figure depending on a language or a culturemay not necessarily be effective.

Moreover, for example, like Japanese and Chinese, even though bothlanguages use the common alphabet (character) called the kanji, the kind(form) of kanji actually used is different. Therefore, the conversationof writing is not necessarily an effective communication means.

Accordingly, in the first embodiment of the present invention, ahandwriting input is enabled in consideration of the validity of theconversation of writing. That is, after handwriting text is translated,the translated text is presented according to orientation (direction)based on the partner's location though a display or a loudspeaker.Moreover, in this case, various kinds of symbols are converted(translated) according to the language and culture of the partner'snative country. In this way, effective communication is enabled.

In FIG. 1, the communication support apparatus 10 comprises at least onepen device 11, an image display unit 12, a magnetic sensor unit 13, acharacter/symbol recognition unit 14, a language/symbol translation unit15, a display information memory 16, a location information memory 17,and a control unit 18.

A pen device 11 (11 a, 11 b, - - - ) outputs a magnetic signalcontaining a peculiar identification signal from a nib for each device.The magnetic sensor unit 13 is set on the image display unit 12. By theuser's holding pen device 11 by hand and tracing on the image displayunit 12 with a nib of the pen device, image information such ascharacters, symbols and figures can be input by handwriting.

The image display unit 12 is consisted of, for example, a liquid crystaldisplay (LCD), and suitably presents a character, a symbol and a figureaccording to an indication of the control unit 18 mentioned later.

By receiving and analyzing the magnetic signal emitted from each pendevice 11, the magnetic sensor unit 13 acquires information such as aspatial position and a locus (movement) of each pen device 11, andinformation about a state of contact on a main part or a posture of thepen device 11. The magnetic sensor unit 13 outputs this pen inputinformation to the control unit 18.

In the first embodiment of the present invention, the control unit 18can detect a relative location relation between a user using the pendevice 11 and the apparatus 10 (the image display unit 12) by referringto the pen input information.

By analyzing the pen input information supplied from the control unit18, the character/symbol recognition unit 14 carries out separationextraction of a character, a symbol, a figure and a drawn image byhandwriting input; acquires information about the locus of a nib, thekind, the contents and a position or an orientation as recognitionresult information; and outputs it to the control unit 18. In addition,the recognition processing of the handwriting input by the pen device isdisclosed in Japanese patent No. 3167799.

In the first embodiment, the control unit 18 detects orientation(direction) of a character/symbol input as the recognition result fromthe character/symbol recognition unit 14, and detects a relativelocation between the user who input the character/symbol as therecognition object by handwriting and apparatus 10 (image display unit12). Moreover, the control unit 18 detects whether the character/symbolis equivalent to which language, and detects a relative locationrelation between the user who input the character/symbol correspondingto the detected language and the apparatus 10 (image display unit 12).

Moreover, the control unit 18 can also decide the user who performed ahandwriting input by analyzing the handwritten input (by analyzing theinformation about the handwriting locus and pressure at the time of thehandwriting input, or the feature of the input image). Concretely, therelative location relation between the user and the apparatus 10 (theimage display unit 12) or the language kind used by the user can bedecided.

The language/symbol translation unit 15 extracts language information ora symbol peculiar to the language kind (culture) from the recognitionresult information supplied from the control unit 18, translates theextracted information by a target language, and outputs the translationresult information to the control unit 18. In addition, the translationprocessing of language information is disclosed in Japanese patent No.3131432.

The control unit 18 acquires various kinds of display information basedon the pen input information, the recognition result information; andthe translation result information. The control unit 18 classifies thevarious kinds of display information suitably, and saves it in thedisplay information memory 16.

FIG. 2 is a schematic diagram showing an example of the contents of thedisplay information memory 16 in FIG. 1. As shown in FIG. 2, the controlunit 18 classifies the input information into ID (identifier)information, position information, orientation information, kindinformation, locus information, translation ID information and contentsinformation, and stores them in the display information memory 16.

As for each entry of the display information memory 16 of FIG. 2, anidentification number of each entry is recorded in a column of the IDinformation.

In a position information column, the position of the area of eachdisplay information on the image display unit 12 is recorded as acoordinate based on the coordinate axis of the image display unit 12. Asshown in FIG. 3, a screen position on the image display unit 12 isrepresented as (X, Y) which is a pair of coordinates, including acoordinate value along an X-axis extended rightward and a coordinatevalue along a Y-axis extended down by making the upper left corner intothe origin. For example, a value L3 of the position information in theentry of a value I3 of the ID information in FIG. 2 is specificallydescribed in FIG. 3.

In the example of FIGS. 2 and 3, the input corresponding to this entryof I3 is a Japanese character shown in FIG. 4. This input is displayedon the position of the coordinates (10, 20) on the display screen.

In an orientation information column, the orientation of each displayinformation is recorded by a vector expression based on the coordinateaxes on the display screen. In addition, a sign “−” is recorded for thedisplay information without the orientation.

In a kind information column, the kind of each display information isrecorded; such as a “figure”, “Japanese”, “English”, a “Japanesesymbol”, and an “English symbol.”

In a locus information column, the input drawing locus corresponding toeach display information acquired from the pen input is recorded. Inaddition, in the entry corresponding to the translation result obtainedby the language/symbol translation unit 15, since the locus informationdoes not exist, the sign “−” is recorded.

Moreover, in a column of the translation ID information, ID of entry ofthe display information which corresponds as a translation pair isrecorded. In addition, a sign “−” is recorded in the entry in which thetranslation pair does not exist.

In a contents information column, information to be displayed on theimage display unit 12 is recorded. In addition, in the case of a figure,a sign “−” is recorded in this column. This shows that the locusinformation should be displayed as it is.

In FIG. 1, the location information memory 17 stores information aboutthe spatial location of each user based on a location of the apparatus10 (image display unit 12).

FIG. 5 is a schematic diagram showing one example of the contents of thelocation information memory 17 in FIG. 1. As shown in FIG. 5,information of the user's location is classified into user IDinformation, location information; and upright orientation information,and is stored in the location information memory 17. As for each entryof the location information memory 17, in a user ID information column,a sign for identifying each user is recorded.

In a location information column, the spatial location relation betweenthe corresponding user and the apparatus 10 is recorded according to thecoordinate system on a screen of the image display unit 12 of theapparatus 10. That is, the relative location of the user for theapparatus 10 is represented by the coordinate axes (having an X-axis anda Y-axis adopted as the coordinate system on a screen of the imagedisplay unit 12) respectively extrapolated along both “+” and “−”directions. For example, when the value of the location information P1of the entry of which user ID is “USER J” in FIG. 5 is (200, −150), asshown in FIG. 6, the user locates at a position corresponding to thecoordinate value “150” along a minus direction of the Y-axis (a dottedline direction on a screen 12 in FIG. 6).

Moreover, in an upright orientation information column, an uprightdirection from the viewpoint of the corresponding user is recorded as avector expression according to the coordinate system on the screen ofthe image display unit 12. This upright orientation is used as a displaydirection at the time of displaying the information for the user.

In FIG. 1, the control unit 18 controls each part of the apparatus 10.Moreover, the control unit 18 analyzes a handwriting input and detectsan input format as the format information of expression used in thehandwriting input. The control unit 18 stores the language kindinformation and the input format information as a group. Therefore, byreferring to the group of stored information, the control unit 18 canalso control an output format of an output message to be the same as theinput format of which the language kind is the same as that of theoutput message.

Next, the operation of the first embodiment of the present invention isexplained with reference to FIG. 7. FIG. 7 is a flow chart of processingsteps executed by the control unit 18.

Here, it is assumed that the entry of each user is prepared in thelocation information memory 17. That is, information about the locationand the orientation of the user based on the apparatus 10 is acquired bythe above-mentioned method and previously recorded in the locationinformation memory 17.

In step S1 of FIG. 7, the control unit 18 is under a standby state ofpen input generating. When a user performs the handwriting input usingthe pen device 11, the magnetic sensor unit 13 detects the position andstate of a nib by operation of the user's pen device 11, and outputsthem to the control unit 18 as pen input information (drawinginformation).

Next, in step S2, the drawing information is stored as displayinformation. That is, after receiving the pen input information, thecontrol unit 18 outputs the drawing information to the character/symbolrecognition unit 14.

In step S3, the character/symbol recognition unit 14 performs languagesymbol recognition processing for the drawing information, and outputsrecognition result information to the control unit 18. After receivingthe recognition result information from the character/symbol recognitionunit 14, the control unit 18 prepares a new entry in the displayinformation memory 16, classifies the recognition result informationsuitably, and records them in the display information memory 16 as shownin FIG. 2. In this case, a sign “−” is recorded in the column of thetranslation ID of the new entry.

In step S4, the control unit 18 retrieves each entry of the displayinformation memory 16 by checking the kind information column and thetranslation ID information. In this case, the entry of which contents ofthe kind information is language (Japanese, English, etc.) or symbol(Japanese symbol, English symbol, etc.) dependent on the language isextracted from the entries each of which the translation ID is “−”.

Hereafter, the control unit 18 repeats processing of step S5-S7 for allextracted entries. That is, the control unit 18 sends the recognitionresult information of the entry that its translation is not recorded tothe language/symbol translation unit 15.

In step S5, the language/symbol translation-unit 15 performs translationprocessing for the recognition result information, and acquires thetranslation result information Rs. The language/symbol translation unit15 outputs the translation result information Rs to the control unit 18.

Next, in step S6, the control unit 18 prepares a new entry Et in thedisplay information memory 16 and records the language kind of thetranslation of the translation result information Rs in the kindinformation column of Entry Et. Moreover, the control unit 18 recordsthe translation itself of the translation result information Rs in thecontents information column of Entry Et.

Next, in step S7, the control unit 18 acquires a value Pt of thelocation information and a value Vt of the upright orientation of a userwho should receive the translation corresponding to the entry Ed byreferring to the location information memory 17. The control unit 18records the value Vt in the upright orientation column of the Entry Et.By referring to the contents of the display information memory 16, thecontrol unit 18 searches (calculates) a position Lt not overlapped withpositions of other entries and near the position of Entry Es, andrecords the position Lt in the position information column of Entry Et.

Next, in step S8, the control unit 18 reads out information of nextentry of the recognition result and repeats processing of steps S4-S8.If each translation of all entries is recorded, the drawing informationis displayed in next step S9. Namely, by referring to the contents ofthe display information memory 16, the control unit 18 outputs thecontents of each entry to the image display unit 12 so that the contentsare displayed at a position where the position information of each entryrepresents and along an upright direction which the orientation of eachentry represents.

Next, a concrete example of usage of the first embodiment is explainedwith reference to the schematic diagram of FIGS. 8 and 9. Explanation ofFIGS. 8 and 9 corresponds to FIG. 2.

In FIG. 8, it is assumed that the first user J who uses Japanese and thesecond user E who uses English face each other while the apparatus 10 islocated between them. Each user mutually explains a place by drawing amap on a display screen 12 of the apparatus 10.

FIG. 9 shows transition of the screen of the image display unit 12 asthe users communicate. Now, as shown in FIG. 9A, the user J inputs adrawing figure I1 showing a river by the pen device 11. In response tothis input, processing of steps S1 and S2 of FIG. 7 is executed, and anentry of ID “I1” is generated in the display information memory 16. Thekind of this entry I1 is a figure. The character/symbol recognition unit14 recognizes that the handwriting input is neither a character nor asymbol. Therefore, in this case, the control unit 18 shifts processingfrom step S4 to step S9 in FIG. 7, and outputs only this figure drawn bythe user as it is to the image display unit 12. In this way, a pictureof the river is displayed on the screen of the image display unit 12.

Next, user J inputs a drawing figure I2 showing a house. In this case,as shown in FIG. 9B, a new entry of I2 is generated in the displayinformation memory 16, and the drawing figure I2 is additionallydisplayed on the screen of the image display unit 12 as it is shown.

Here, it is assumed that the user J inputs the Japanese character shownin FIG. 4 near the drawing figure showing a house. In this case, newentry of I3 is generated in the display information memory 16 by thecontrol unit 18. The contents of the entry I3 is the languageinformation which can be translated, and new entry I4 of “HOME” inEnglish (as a translation of the contents of the entry I3) is generatedin the display information memory 16 according to the recognition resultof the character/symbol recognition unit 14. Moreover, the control unit18 sets a display position of the entry I4 on the screen so that thedisplay position is near the Japanese I3 and a location of the user E,not overlapped with display positions of other characters/symbols I1,I2, I3, and an orientation of the character I4 is upright orientationfrom the user E's view point (Steps S5-S8).

In step S9, based on a setup of the control unit 18, as shown in FIG.9C, Japanese character of FIG. 4 inputted by the user J is displayed inaddition to the previous contents of the screen. Furthermore, as shownin FIG. 9D, the English character string “HOME”, which is a translationof Japanese character I3, is displayed near the Japanese character I3along the upright orientation from the user E's view point.

Subsequently, as shown in FIG. 9E, it is assumed that the user E inputsa drawing figure I5 showing a bridge and an English character string I6“bridge” by handwriting. The above-mentioned processing is performed inthe same way and a translation character string I7 in Japanese shown inFIG. 10A is displayed along the upright orientation from the user j'sview point.

Next, it is assumed that the user J inputs a symbol of a post office I8shown in FIG. 10B by handwriting. In this case, the character/symbolrecognition unit 14 extracts a drawing pattern of handwriting as asymbol dependent on language culture. Furthermore, the language/symboltranslation unit 15 obtains the translation result I9 as Englishcharacter string “post office” for this symbol. The control unit 18additionally displays this character string I9 as an annotation on thescreen along the upright orientation from the user E's view point (FIG.9 F).

Thus, in the first embodiment, it is decided whether a user'shandwriting message is a character or a symbol (generally referred to aslanguage dependence data), and translation processing for a user of theother party is performed not only for a character but for a symbol.Furthermore, a translation result is presented as a message along theupright orientation from this user's viewpoint. Since a conversation bywriting using not only a character but a symbol or a picture is adopted,even if the number of recorded expressions for communications (such as afixed form phrase) is few, it is possible to provide improvedinformation necessary for communication with a partner. The input by thehandwriting of a character or a picture is possible, and input operationis very easy. That is, since complex translation processing with apicture, a symbol and a character is performed, communication using apicture (image) is extremely simplified and it is effective inexplanation of a place, etc. Furthermore, a symbol which is notoriginally the object of translation can be translated as a character.As a result, the other user as a partner can easily understand a messagefrom a user.

Furthermore, in the first embodiment, a presentation position anddirection of a message are determined according to each user's location.Therefore, the translation result of the character/symbol written by auser and a partner can be simultaneously seen along the uprightorientation for both the user and the partner, and communication can beeffectively supported.

For example, it is assumed that a plurality of users commonly use onedisplay screen, and each of the plurality of users performs ahandwriting input using each pen device on the display screen. Forexample, the first user who speaks Japanese additionally writes anannotation in Japanese beside a certain figure drawn by the first user.The annotation is translated into English for the second user who speaksEnglish, and the translated result is displayed side by side with theoriginal annotation of Japanese. In this case, the translated annotationof English is displayed at a position physically near the second user onarea neighboring the certain figure of the screen.

Moreover, it is assumed that the first user who speaks Japanese and thesecond user who speaks English face each other by locating a deskbetween them, and they use one display screen evenly put on the desk.When the first user additionally writes an annotation of Japanese besidea certain figure drawn by the first user, a translation result ofEnglish of the original annotation is displayed for the second user. Inthis case, the English annotation is displayed toward the uprightorientation from the second user's viewpoint on area neighboring thecertain figure of the screen.

Moreover, in the case that the input by handwriting is received from auser, for example, orientation of the input character/figure or drawingdirection of input stroke is analyzed, and the spatial location of theuser who inputted by handwriting is detected and stored. Based on thisinformation, a presentation method of the message to present to eachuser is controlled, and the character information message is presentedto the user. In this case, by referring to the location information ofthe user detected by above-mentioned processing, the message isdisplayed along the upright orientation from the user's viewpoint.

Moreover, in the case that the handwriting input is received from auser, the kind of input character is analyzed, and the language kind ofthe user is decided. The language kind information is utilized tocontrol the output format for the user. For example, by analyzing thehandwritten input from the first user, it is decided that a character inthe input is Japanese and an upright orientation of character in theinput is which direction on the screen. Next, the input in English fromthe second user is translated into Japanese and presented to the firstuser. In this case, a message of Japanese as a translation result ofinput (English) from the second user is displayed along the samedirection as the upright orientation of the handwritten input (Japanese)from the first user decided by the above-mentioned processing.

Moreover, for example, information that the first user uses Japanese andthe second user uses English may be stored. That is, when an input isreceived from a user, the user's language kind is decided by analyzingthe input and stored in correspondence with spatial location of theuser. In this way, the language kind can be utilized to control thepresentation of a message afterwards.

In the first embodiment, the image display unit preparing a pen deviceand a magnetic sensor is commonly used as an input unit and apresentation unit. However, the same communication support is possibleby sound using a microphone and a loudspeaker to input and output soundrespectively.

FIG. 11 is a block diagram of the communication support apparatusaccording to the second embodiment of the present invention, and thisshows the apparatus capable of communicating support with sound. In FIG.11, the same labels are given to the same components as FIG. 1, andexplanation of the same components are omitted.

In addition to the components shown in FIG. 1, the communication supportapparatus 20 of the second embodiment further includes microphones 21 aand 21 b, loudspeakers 22 a and 22 b, an acoustic recognition unit 25,an acoustic translation unit 26, and an acoustic output unit 27.

FIG. 12 is a schematic diagram for explaining the example arrangement ofmicrophones 21 a and 21 b and loudspeakers 22 a and 22 b. As shown inFIG. 12, a pair of microphones 21 a and 21 b is attached to both ends ofone side of the image display unit 12, and a pair of loudspeakers 22 aand 22 b is attached to both ends of the other side of the image displayunit 12.

The microphones 21 a and 21 b output a sound signal to the control unit18. The microphones 21 a and 21 b are located a predetermined distancefrom each other. By a time delay relation of the sound signal from eachmicrophones 21 a and 21 b, the control unit 18 can detect a location ofthe user who uttered the sound signal taken in by the microphones 21 aand 21 b.

The control section 18 gives the sound signal from the microphones 21 aand 21 b to the acoustic recognition unit 25. The acoustic recognitionunit 25 recognizes the inputted sound signal, and returns a speechrecognition result to the control unit 18. The control unit 18 outputsthe speech recognition result to the acoustic translation unit 26. Theacoustic translation unit 26 translates the speech recognition result,and returns a translation result to the control unit 18. The controlunit 18 outputs the translation result from the acoustic translationunit 26 to the acoustic output unit 27.

The acoustic output unit 27 includes a sound source allocation apparatus28. The acoustic output unit 27 converts the translation result into asound signal. The sound source allocation apparatus 28 adjusts a level,a sound quality and a phase of the sound signal so that the sound signalof the translation result is allocated towards a user whom the soundsignal is presented. In this way, the acoustic output unit 27 outputs anadjusted sound signal to each of the loudspeakers 22 a and 22 b.

In addition, the number of microphones and the number of loudspeakersare not limited to two. At least three microphones and at least threeloudspeakers may also be prepared.

In addition, the control unit 18 may analyze the speech signal, andextract information about the voiceprint and utterance style acquiredfrom the speech signal. From such extracted information, it is alsopossible to identify the user who uttered the sound related to thespeech signed.

Thus, in the second embodiment, not only a handwriting input but aspeech input can be applied. A language by this speech input istranslated into another language that a partner can recognize. Thesystem is then able to output the translated sound signal towards alocation of the partner.

Moreover, if the control unit 18 gives the recognition result of ahandwriting input to the acoustic translation unit 26, it is possiblethat a text by the handwriting input can be output as speech using apartner's language toward the partner. Conversely, if the control unit18 gives the recognition result of a speech input to the language/symboltranslation unit 15, it is possible that the speech input is convertedto a text using a partner's language and output on a screen of the imagedisplay unit 12.

Thus, in the second embodiment, a speech uttered by a user is convertedto a speech using a partner's understandable language, and the convertedspeech is output toward the partner's location. Accordingly, efficiencyof communication remarkably improves.

For example, it is assumed the first user who speaks Japanese and thesecond user who speaks English respectively locate at the right side andthe left side of one common display. In this case, the speech signal inJapanese to be output to the first user is controlled so that the volumebecomes large on the right side as it is the first user's location. Inthe same way, the speech signal in English to be output to the seconduser is controlled so that the volume becomes large on the left side asit is the second user's location.

For example, a plurality of microphones is prepared. By using a timedelay relation of the speech signal acquired from each microphone,spatial location from which each speech signal is outputted can bedetected (this is commonly called the microphone array technique). Inshort, a spatial location relation between each user and the apparatus20 can be detected. Based on the location information, it is possiblethat a character/picture message is displayed at a screen position neara user who should receive the message at an upright orientation from theuser's view point. Moreover, it is possible that a message of speech isallocated or projected towards a location of a user who should receivethe message.

Furthermore, after receiving a speech input from a user, by analyzingthe speech input, the user who uttered the speech input is identified.By utilizing this information, a spatial location relation between eachuser and the apparatus 20 is analyzed and used for controlling outputformat for the user.

The present invention is not limited to the above-mentioned embodiments.For example, spatial position and spatial posture of a pen device can beanalyzed by embedding a magnetic tag into the pen device used for ahandwriting input and by analyzing the tag information by the magneticsensor. In this way, it is also possible to detect the spatial locationrelation between this apparatus and a user.

Moreover, each user using this apparatus may put on a wireless tag tothe body, and the wireless tag information can be read by an antennawith directivity and analyzed. In this way, it is also possible todetect the spatial location relation between this apparatus and a user.

Moreover, it is not always necessary that an input is made in real timefrom a user. The input may be provided through a medium or a network.

Moreover, a plurality of input means such as a handwriting input and aspeech input may be simultaneously used together.

Moreover, for example, an input may be executed by a menu selection.

Moreover, the type of input means and the type of output means may bedifferent. For example, the input can be executed by speech and theoutput can be executed by character image.

Moreover, the input means for each user may be different. For example,one user may input by speech and another user may input by handwriting.

Moreover, the output means for each user may be different. For example,an output can be executed by speech to one user and the output can beexecuted by character image to another user.

Moreover, in addition to a function to translate language, a mediaconversion function such as conversion from a sound signal to acharacter signal may be commonly used.

Moreover, an output may be executed to a plurality of userssimultaneously.

Moreover, there may be a plurality of input means and output means.

Moreover, in the above-mentioned embodiments, one apparatus commonlyused by a plurality of users is explained. However, a plurality ofapparatuses for communicating with each other may be adopted inaccordance with an alternative embodiment of the present invention.

Moreover, a decision result of language kind may be utilized fordetermining a translation direction from one language to anotherlanguage.

Moreover, a decision result of an input user may be utilized fordetermining the translation direction as a language king of the inputuser.

Moreover, when this apparatus 10 (20) is realized as a portableequipment which a user can have by hand, by preparing a means fordetecting a spatial posture of a main part, an output format may becontrolled using the posture information together.

Moreover, in the above-mentioned embodiments, the orientation of thecharacter is controlled on the screen. However, for example, translucentdisplay, multicolor display, or stereoscopic display, may be adopted.

Moreover, language used by each user uses is not limited to one. Aplurality of languages may be used by each user. The language which auser uses may be set up beforehand or the user may set the use languagesuitably.

Moreover, in a processing for detecting the location relation betweeneach user and the apparatus, an antipersonnel sensor, an imagerecognition technique for a face or a hand, or a motion capturetechnique, may be used.

Moreover, in the case of controlling the output format of the speechsignal, at a location of a user who should not receive the speechsignal, the speech signal may become small or may be masked not to hearfor the user.

As mentioned-above, in the present invention, smooth communication for aplurality of persons each using a different language is possible.

For embodiments of the present invention, the processing of thecommunication support apparatus of the present invention can beaccomplished by computer-executable program, and this program can berealized in a computer-readable memory device.

In embodiments of the present invention, the memory device, such as amagnetic disk, a floppy disk, a hard disk, an optical disk (CD-ROM,CD-R, DVD, and so on), an optical magnetic disk (MD, and so on) can beused to store instructions for causing a processor or a computer toperform the processes described above.

Furthermore, based on an indication of the program installed from thememory device to the computer, OS (operation system) operating on thecomputer, or MW (middle ware software), such as database managementsoftware or network, may execute one part of each processing to realizethe embodiments.

Furthermore, the memory device is not limited to a device independentfrom the computer. By downloading a program transmitted through a LAN orthe Internet, a memory device in which the program is stored isincluded. Furthermore, the memory device is not limited to one. In thecase that the processing of the embodiments is executed by a pluralityof memory devices, a plurality of memory devices may be included in thememory device. The component of the device may be arbitrarily composed.

In embodiments of the present invention, the computer executes eachprocessing stage of the embodiments according to the program stored inthe memory device. The computer may be one apparatus such as a personalcomputer or a system in which a plurality of processing apparatuses areconnected through the network. Furthermore, in the present invention,the computer is not limited to the personal computer. Those skilled inthe art will appreciate that a computer includes a processing unit in aninformation processor, a micro computer, and so on. In short, theequipment and the apparatus that can execute the functions inembodiments of the present invention using the program are generallycalled the computer.

Other embodiments of the invention will be apparent to those skilled inthe art from consideration of the specification and practice of theinvention disclosed herein. It is intended that the specification andexamples be considered as exemplary only, with the true scope and spiritof the invention being indicated by the following claims.

What is claimed is:
 1. A communication support apparatus, comprising: acommon display unit; a plurality of microphones located a predetermineddistance from each other and attached to one side of the common displayunit, configured to receive a first input message from a first user anda second input message from a second user, the first user and the seconduser respectively having a first spatial location and a second spatiallocation relative to the common display unit; a plurality ofloudspeakers attached to the other side of the common display unit; acontrol unit configured to decide the first spatial location by a timedelay relation of the first input message from each of the microphones,and to decide the second spatial location by a time delay relation ofthe second input message from each of the microphones; and a translationunit configured to translate the first input message based on a firstlanguage of the first user into a first output message based on a secondlanguage of the second user, and to translate the second input messagebased on the second language into a second output message based on thefirst language; wherein said control unit is configured to select afirst loudspeaker based on the decided first spatial location from theplurality of loudspeakers, to select a second loudspeaker based on thedecided second spatial location from the plurality of loudspeakers, toadjust the first output message so that the adjusted first outputmessage is allocated towards the decided second spatial location via thesecond loudspeaker, and to adjust the second output message so that theadjusted second output message is allocated towards the decided firstspatial location via the first loudspeaker.
 2. The communication supportapparatus according to claim 1, wherein the first input message, thesecond input message, the first output message, and the second outputmessage are sound signals containing at least one of a spoken languageand non-verbal sound.
 3. The communication support apparatus accordingto claim 2, wherein the microphones input acoustic data uttered by thefirst user as the first input message and by the second user as thesecond input message.
 4. The communication support apparatus accordingto claim 3, wherein the control unit detects a language kind of each ofthe first user and the second user in accordance with features of thefirst input message and the second input message from each of the firstuser and the second user.
 5. The communication support apparatusaccording to claim 4, wherein said control unit decides the languagekind in accordance with information about a glottis or an utterancestyle acquired from spoken languages of the first input message and thesecond input message.
 6. The communication support apparatus accordingto claim 5, further comprising: a location information memory configuredto store the first spatial location, the second spatial location, andthe language kind of each of the first user and the second user.
 7. Thecommunication support apparatus according to claim 6, wherein saidtranslation unit retrieves the language kind of the first user and thesecond user from said location information memory, translates the firstinput message based on the first language into the first output messagebased on the second language, and translates the second input messagebased on the second language into the second output message based on thefirst language.
 8. The communication support apparatus according toclaim 7, wherein said control unit retrieves the first spatial locationand the second spatial location from said location information memory.9. A method for supporting communication in a computer connected with acommon display unit, a plurality of microphones and a plurality ofloudspeakers, the plurality of microphones being located a predetermineddistance from each other and attached to one side of the common displayunit, the plurality of loudspeakers being attached to the other side ofthe common display unit, comprising: receiving a first input messagefrom a first user and a second input message from a second user via theplurality of microphones, the first user and the second userrespectively having a first spatial location and a second spatiallocation relative to the common display unit; deciding the first spatiallocation by a time delay relation of the first input message from eachof the microphones; deciding the second spatial location by a time delayrelation of the second input message from each of the microphones;translating the first input message based on a first language of thefirst user into a first output message based on a second language of thesecond user; translating the second input message based on the secondlanguage into a second output message based on the first language;selecting a first loudspeaker based on the decided first spatiallocation from the plurality of loudspeakers; selecting a secondloudspeaker based on the decided second spatial location from theplurality of loudspeakers; adjusting the first output message so thatthe adjusted first output message is allocated towards the decidedsecond spatial location via the second loudspeaker; and adjusting thesecond output message so that the adjusted second output message isallocated towards the decided first spatial location via the firstloudspeaker.
 10. A computer program product, encoded in a non-transitorycomputer readable medium comprising: a computer readable program codeembodied in said product for causing a computer to supportcommunication, the computer being connected with a common display unit,a plurality of microphones and a plurality of loudspeakers, theplurality of microphones being located a predetermined distance fromeach other and attached to one side of the common display unit, theplurality of loudspeakers being attached to the other side of the commondisplay unit, said computer readable program code having: a firstprogram code to receive a first input message from a first user and asecond input message from a second user via the plurality ofmicrophones, the first user and the second user respectively having afirst spatial location and a second spatial location relative to thecommon display unit; a second program code to decide the first spatiallocation by a time delay relation of the first input message from eachof the microphones; a third program code to decide the second spatiallocation by a time delay relation of the second input message from eachof the microphones; a fourth program code to translate the first inputmessage based on a first language of the first user into a first outputmessage based on a second language of the second user; a fifth programcode to translate the second input message based on the second languageinto a second output message based on the first language; a sixthprogram code to select a first loudspeaker based on the decided firstspatial location from the plurality of loudspeakers; a seventh programcode to select a second loudspeaker based on the decided second spatiallocation from the plurality of loudspeakers; an eighth program code toadjust the first output message so that the adjusted first outputmessage is allocated towards the decided second spatial location via thesecond loudspeaker; and a ninth program code to adjust the second outputmessage so that the adjusted second output message is allocated towardsthe decided first spatial location via the first loudspeaker.